{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image tag assignment by TagVote\n",
    "\n",
    "An example showing how to do tag assignment by the TagVote method, using train10k as training data and mirflickr08 as test data.\n",
    "\n",
    "\n",
    "## Prepare \n",
    "\n",
    "* Download vggnet features\n",
    "```\n",
    "cd $HOME/VisualSearch\n",
    "wget http://lixirong.net/data/csur2016/train10k-vggnet16-fc7relu.tar.gz\n",
    "wget http://lixirong.net/data/csur2016/mirflickr08-vggnet16-fc7relu.tar.gz\n",
    "```\n",
    "\n",
    "* Download tag data of train10k\n",
    "```\n",
    "wget http://lixirong.net/data/csur2016/train10k-tag.tar.gz\n",
    "```\n",
    "\n",
    "* Download annotation files of mirflickr08\n",
    "```\n",
    "wget http://lixirong.net/data/csur2016/mirflickr08-anno.tar.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code\n",
    "\n",
    "### Create a TagVote instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "09/05/2016 14:15:10 INFO [/Users/xirong/github/jingwei/instance_based/tagvote.pyc] TagVoteTagger, 10000 images, 41253 unique tags, cosine 1000 neighbours for voting\n"
     ]
    }
   ],
   "source": [
    "from instance_based.tagvote import TagVoteTagger\n",
    "\n",
    "trainCollection = 'train10k'\n",
    "annotationName = 'concepts130.txt'\n",
    "feature = 'vgg-verydeep-16-fc7relu'\n",
    "\n",
    "tagger = TagVoteTagger(collection=trainCollection, annotationName=annotationName, feature=feature, distance='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open feature file of mirflickr08"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[BigFile] 25000x4096 instances loaded from /Users/xirong/VisualSearch/mirflickr08/FeatureData/vgg-verydeep-16-fc7relu\n"
     ]
    }
   ],
   "source": [
    "from basic.constant import ROOT_PATH\n",
    "from util.simpleknn.bigfile import BigFile\n",
    "import os\n",
    "\n",
    "rootpath = ROOT_PATH\n",
    "testCollection = 'mirflickr08'\n",
    "feat_dir = os.path.join(rootpath, testCollection, 'FeatureData', feature)\n",
    "feat_file = BigFile(feat_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load image ids of mirflickr08"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# load image ids of mirflickr08\n",
    "from basic.util import readImageSet\n",
    "testimset = readImageSet(testCollection)\n",
    "\n",
    "# load a subset of 200 images for test\n",
    "import random\n",
    "testimset = random.sample(testimset, 200)\n",
    "renamed, vectors = feat_file.read(testimset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform tag relevance learning on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing 200 images took 14.717 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "s_time = time.time()\n",
    "results = [tagger.predict(vec) for vec in vectors]\n",
    "timespan = time.time() - s_time\n",
    "print ('processing %d images took %g seconds' % (len(renamed), timespan))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation \n",
    "\n",
    "First, we need to load ground-truth of mirflickr08, which is provided at the folder *$HOME/VisualSearch/mirflickr08/Annotations*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baby has 116 positives\n",
      "bird has 484 positives\n",
      "car has 380 positives\n",
      "cloud has 1350 positives\n",
      "dog has 590 positives\n",
      "flower has 1077 positives\n",
      "girl has 3982 positives\n",
      "man has 3647 positives\n",
      "night has 669 positives\n",
      "people has 7849 positives\n",
      "portrait has 3829 positives\n",
      "river has 149 positives\n",
      "sea has 214 positives\n",
      "tree has 668 positives\n"
     ]
    }
   ],
   "source": [
    "from basic.annotationtable import readConcepts, readAnnotationsFrom\n",
    "\n",
    "testAnnotationName = 'conceptsmir14.txt'\n",
    "concepts = readConcepts(testCollection, testAnnotationName)\n",
    "nr_of_concepts = len(concepts)\n",
    "\n",
    "label2imset = {}\n",
    "im2labelset = {}\n",
    "\n",
    "for i,concept in enumerate(concepts):\n",
    "    names,labels = readAnnotationsFrom(testCollection, testAnnotationName, concept)\n",
    "    pos_set = [x[0] for x in zip(names,labels) if x[1]>0]\n",
    "    print ('%s has %d positives' % (concept, len(pos_set)))\n",
    "    for im in pos_set:\n",
    "        label2imset.setdefault(concept, set()).add(im)\n",
    "        im2labelset.setdefault(im, set()).add(concept)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute map based on image ranking results\n",
    "\n",
    "1. For each test concept, sort the test images in descending order according to their relevance scores with respect to the concept\n",
    "2. Compute Average Precision of the concept\n",
    "3. Compute mean Average Precision, by averaging AP scores of the concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baby 0.000\n",
      "bird 0.561\n",
      "car 0.917\n",
      "cloud 0.434\n",
      "dog 0.679\n",
      "flower 0.673\n",
      "girl 0.500\n",
      "man 0.597\n",
      "night 0.426\n",
      "people 0.795\n",
      "portrait 0.723\n",
      "river 0.333\n",
      "sea 0.292\n",
      "tree 0.589\n",
      "meanAP 0.537\n"
     ]
    }
   ],
   "source": [
    "# sort images to compute AP scores per concept\n",
    "ranklists = {}\n",
    "for _id, res in zip(renamed,results):\n",
    "    for tag,score in res:\n",
    "        ranklists.setdefault(tag, []).append((_id, score))\n",
    "\n",
    "from basic.metric import getScorer\n",
    "scorer = getScorer('AP')\n",
    "\n",
    "mean_ap = 0.0\n",
    "for i,concept in enumerate(concepts):\n",
    "    pos_set = label2imset[concept]\n",
    "    ranklist = ranklists[concept]\n",
    "    ranklist.sort(key=lambda v:(v[1], v[0]), reverse=True) # sort images by scores in descending order\n",
    "    sorted_labels = [2*int(x[0] in pos_set)-1 for x in ranklist]\n",
    "    perf = scorer.score(sorted_labels)\n",
    "    print ('%s %.3f' % (concept, perf))\n",
    "    mean_ap += perf\n",
    "mean_ap /= len(concepts)\n",
    "print ('meanAP %.3f' % mean_ap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute miap based on tag ranking results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "miap 0.365\n"
     ]
    }
   ],
   "source": [
    "# compute iAP per image\n",
    "miap = 0.0\n",
    "for _id, res in zip(renamed,results):\n",
    "    pos_set = im2labelset.get(_id, set()) # some images might be negatives to all the 14 concepts\n",
    "    ranklist = [x for x in res if x[0] in label2imset] # evaluate only concepts with ground truth\n",
    "    sorted_labels = [2*int(x[0] in pos_set)-1 for x in ranklist]\n",
    "    perf = scorer.score(sorted_labels)\n",
    "    miap += perf\n",
    "miap /= len(renamed)\n",
    "print ('miap %.3f' % miap)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
